Spatially-Aware Tool System

ABSTRACT

The systems described in this disclosure can be used in construction settings to facilitate the tasks being performed. The location of projectors and augmented reality headsets can be calculated and used to determine what images to display to a worker, based on a map of work to be performed, such as a construction plan. Workers can use spatially-aware tools to make different locations be plumb, level, or equidistant with other locations. Power to tools can be disabled if they are near protected objects.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.17/499,469, filed Oct. 12, 2021, which is a continuation of U.S.application Ser. No. 16/682,955, now U.S. Pat. No. 11,170,538, filedNov. 13, 2019, which is a continuation of International Application No.PCT/US2019/058757, filed Oct. 30, 2019, which claims the benefit of andpriority to U.S. Provisional Application No. 62/753,389, filed Oct. 31,2018, which are incorporated herein by reference in their entireties.

BACKGROUND OF THE INVENTION

The present disclosure relates generally to the field of constructiontools. Construction work generally requires non-digital hand tools suchas tape measures, levels, and power tools that do not calculate theirlocation. A worker refers to a construction plan, such as a blueprint,to identify work that needs to be performed.

SUMMARY OF THE INVENTION

In general, an embodiment of the disclosure relates to tracking thephysical position of a spatially-aware tool, associating the physicalposition with a construction plan, generating augmentation data based onthe association, and providing the augmentation data to the user. Aprojector displays an image based on the augmentation data representingthe construction plan on the floors and walls in the room where work isto be performed. The projector positions the virtual objects in theimage onto the physical locations where the physical objects will belocated according to the construction plan.

The projector includes a toggle to allow a worker to switch thedisplayed image to a schematic of the construction plan. Workersfamiliar with reading construction plans can look at the projected imagein a room and immediately understand what work has to be performed inwhich parts of the room.

When generating the image based on the augmentation data, the projectoridentifies the physical landscape of walls, ceiling(s) and floor(s) thatthe image will be projected on. The projector then generates the imagebased on the relationship between the physical landscape available tothe projector and what objects in the construction plan will beinstalled in/on the physical landscape.

An augmented reality (AR) headset receives the augmented data andgenerates an augmented image to display to the worker wearing the ARheadset. The augmented image includes a combination of the physicalimage in front of the AR headset and the worker, and a virtual image ofobjects from a construction plan. Images of the virtual objects areplaced at locations in the augmented image that correspond to where thecorresponding physical objects should be placed according to theconstruction plan.

In one embodiment, the location of a spatially-aware drill is comparedagainst a map of delicate or dangerous objects, such as an electricalwire or water pipe that should be protected. The location of theprotected object is based on a construction plan, or it may be based onan as-built model that reflects where object was actually installed. Adistance between the drill bit tip and the protected object is monitoredand compared to a threshold. When the distance is less than thethreshold, power to the drill is disabled and the drill generates asignal to display to the worker why the drill's power was disabled.

In one embodiment, a spatially-aware tool, such as a drill, is used todefine an artificial frame of reference. First, a component of the drillis calibrated to the room, such as by calibrating the location of thedrill bit tip by placing it against a spatially-tracked sensor. Thetracked location of the drill bit tip is placed at an origin of theartificial frame of reference, such as a corner between two walls and afloor, and a signal is generated to identify the origin. The drill bittip is then placed at another corner between one of the walls and thefloor and a second signal is generated to identify the first axis. Thefirst axis is defined by connecting to the identified points.

The worker identifies special points, such as where to drill a hole, inthe spatially-aware room with respect to the artificial frame ofreference. The artificial frame of reference and the special points arecommunicated to another worker with an AR headset. The AR headset of thesecond worker displays an augmented image that shows the special pointsindicated against the wall. Thus, the second worker can visualize thespecial points and where to drill the hole(s).

The physical location of a spatially-aware tool with a first worker istracked by the system and sent to a remote worker. The remote workerviews a virtual image of the construction plan at the physical location.The spatially-aware tool may include a camera that captures a physicalimage that is streamed to the remote worker. The physical image can becombined with the virtual image to create an augmented image. Theaugmented image is displayed for the second worker, such as via an ARheadset or a projector.

In one embodiment, a worker wearing an AR headset is shown a menu ofvirtual items that can be installed. The worker selects from amongdifferent categories of items to install, such as HVAC (heating,ventilation, and air conditioning) or plumbing. The worker selectscomponents to install in the room from the menu. The worker virtuallyinstalls the selected component by indicating the location of theselected component. The worker continues selecting components andinstalling them in and near the room. After the virtual construction iscomplete, the worker can indicate that the selected components need tobe ordered and delivered to that room. In some instances, actualcomponents can be fabricated remotely based on the virtual installationand delivered to the room for installation.

In various embodiments, the system may incorporate a projector on atripod, an AR flashlight, a lighthouse, a sensor block, or a staff, allof which are spatially-aware and thus system-enabled. These devices maywork in coordination with each other to identify points where workshould be performed. These devices can rely on each other to determinetheir location. The location determination of a device may be arespective location with respect to an artificial frame of reference, orit may be an absolute location with respect to a construction plan. Asthe devices determine their location based on other devices, the devicescan leapfrog across an area to provide spatially-aware guidance for thearea.

Additional features and advantages will be set forth in the detaileddescription which follows, and, in part, will be readily apparent tothose skilled in the art from the description or recognized bypracticing the embodiments as described in the written description andclaims hereof, as well as the appended drawings. It is to be understoodthat both the foregoing general description and the following detaileddescription are exemplary.

The accompanying drawings are included to provide further understandingand are incorporated in and constitute a part of this specification. Thedrawings illustrate one or more embodiments and, together with thedescription, serve to explain principles and operation of the variousembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top view of a system-enabled room with lighthouses, aprojector, and an AR headset, according to an embodiment.

FIG. 2 is a perspective view of the system-enabled room of FIG. 1 , theprojector showing a perspective-based augmented reality, according to anembodiment.

FIG. 3 is a perspective view of the system-enabled room of FIG. 1 , theprojector showing a schematic-based augmented reality, according to anembodiment.

FIG. 4 is an augmented reality perspective view of a system-enabled roomwith two lighthouses and an AR headset, according to an embodiment.

FIG. 5 is an augmented reality perspective view of a system-enabled roomwith two lighthouses and an artificial frame of reference against onewall, according to an embodiment.

FIG. 6 is a virtual perspective view of a system-enabled room, accordingto an embodiment.

FIG. 7 is a perspective view of the system-enabled room of FIG. 6 ,according to an embodiment.

FIG. 8 is an augmented reality perspective view of the system-enabledroom of FIG. 6 , according to an embodiment.

FIG. 9 is an augmented reality perspective view of a system-enabled roomwith multiple locations identified and displayed, according to anembodiment.

FIG. 10 is an augmented reality perspective view of a system-enabledroom with a window and multiple locations identified and displayed,according to an embodiment.

FIG. 11 is an augmented reality perspective view of a system-enabledroom with multiple artificial objects identified and displayed,according to an embodiment.

FIG. 12 is a perspective view of a system-enabled tripod with aprojector, according to an embodiment.

FIG. 13 is a perspective view of a system-enabled flashlight, accordingto an embodiment.

FIG. 14 is a perspective view of a system-enabled lighthouse, accordingto an embodiment.

FIG. 15 is a perspective view of a system-enabled sensor block,according to an embodiment.

FIG. 16 is a perspective view of a system-enabled augmented reality (AR)headset, according to an embodiment.

FIG. 17 is a perspective view of a system-enabled staff, according to anembodiment.

FIG. 18 is a perspective view of a total robotic system, according to anembodiment.

FIG. 19 is a perspective view of a total robotic system, according to anembodiment.

FIG. 20 is a perspective view of a tripod with an articulating laser,according to an embodiment.

FIG. 21 is a perspective view of a laser receiver with vertical linelaser, according to an embodiment.

FIG. 22 is a perspective view of a multi-direction camera system,according to an embodiment.

FIGS. 23A-23C are a series of top views of a room with a tripod with anarticulating laser synchronizing by interacting with a robotic station,according to an embodiment.

FIGS. 24A-24D are a series of top views of a room with a tripod with anarticulating laser and a beacon, according to an embodiment.

FIGS. 25A-25C are a series of top views of a room with multiple sensorsand a lighthouse, according to an embodiment.

FIGS. 26A-26C are a series of top views of a room with multiplelighthouses and a personal sensor, according to an embodiment.

FIGS. 27A-27C are a series of top views of a room with multiple infraredemitters and a personal sensor, according to an embodiment.

FIGS. 28A-28E are a series of top views of a room with multiple infraredreceivers and a digital lighthouse, according to an embodiment.

FIGS. 29A-29B are a series of top views of a room with alocation-detecting projector with non-continuous refresh, according toan embodiment.

FIGS. 30A-30B are a series of top views of a room with alocation-detecting flashlight projector with continuous refresh,according to an embodiment.

FIG. 31 is a top view of a room with multiple lighthouses, knownlocations, and personal sensors, according to an embodiment.

FIGS. 32A-32F are a series of top views of a room with sensorsleapfrogging across a room, according to an embodiment.

FIGS. 33A-33D are a series of top views of a room with lighthousesleapfrogging across a room, according to an embodiment.

FIG. 34 is an augmented reality perspective view of a room with an ARheadset including stereo cameras, according an embodiment.

FIG. 35 is an augmented reality perspective view of objects as built ina room, according to an embodiment.

DETAILED DESCRIPTION

Referring generally to the figures, various embodiments of usingspatially-aware tools are shown. The systems described in thisdisclosure can be used in construction settings to vastly facilitate thetasks being performed.

In a system-enabled room, in which the location of tools and displayscan be calculated, the system provides images based on a constructionplan to help workers identify where different components or objects arebeing installed. The system displays these images by using a projectorthat displays the images on walls, floors and ceilings in the room, orthe system can provide these images by augmented reality (AR) headsets.The images provided can be actual images of objects to be built andinstalled, or the images can be representative symbols, such as may befound in a blueprint, that allow workers to quickly understand theconstruction plans in the image being projected.

The system provides more accurate measurements that are easier toacquire than by using non spatially-aware tools, such as hand tools.Workers can use spatially-aware tools in place of levels and tapemeasures. The spatially-aware tools, such as a drill or an AR headset,provide feedback to the user about different points being identified.Based on that information, the worker can make the points plumb, level,equidistant with other points, or any other arrangement that would haveotherwise required a level or tape measure.

A supervisor can identify a wall as a local artificial frame ofreference, and identify points on that wall where work is to beperformed. This identification of points can be performed by thesupervisor without reference to a construction plan. The points can besaved to the system, and another worker can walk through the rooms andperform the work identified by the supervisor by relying on the localartificial frame(s) of reference.

The system can disable tools before they cause damage. For example, if adrill bit is about to puncture a hole in a delicate or dangerous object(e.g., electrical wire, water pipe), the system can deactivate power tothe drill's motor that rotates the bit and prevent damaging theprotected object but retain power for the remaining systems and thedrill, allowing the user to receive a message or indicationcommunicating why the drill bit was stopped.

Referring FIGS. 1-3 , a tracking system, such as a light-based system,is shown according to an exemplary embodiment. System 40 includeslighthouses 42, projector 44, and augmented reality (AR) headset 46.Lighthouses 42 emit a pattern of light across a room that receivers 120on device 70 can use to determine their position. Device 70 includesspatially-aware devices in a system-enabled room, such as lighthouse 42,projector 44, AR headset 46, sensor 48, tool 72, tripod 110, flashlightprojector 116, staff 122, total robotic station 140, articulating laser142, laser receiver with vertical line laser 146, and multi-directioncamera system 148.

Projector 44 and AR headset 46 include one or more receivers 120 todetermine their location and orientation, collectively position 52.Projector 44 and AR headset 46 use their location and orientationinformation to determine images 54 to display for a user.

In various embodiments, device 70 determines its location based ondetecting light from lighthouses 42. In alternative embodiments, one ormore of the computer processors of the system 40 may determine thelocation of the device 70. This position determination analysis may besupplemented by internal measuring units (IMUs), such as kinetic sensorsincluding accelerometers. Lighthouse 42 initially emits a pulse of lightand then emits a series of light bands, such as calibration signals 150,in a pattern across the room. Device 70 detects a band and measures thetime period between the initial pulse of light and the detected band.Based on the length of the time period measured and known position 52 oflighthouse 22, device 70 determines first relative position 50 of device70 with respect to first lighthouse 42. Device 70 similarly determinessecond relative position 50 of device 70 with respect to secondlighthouse 42. Based on the combination of the relative positions 50,device 70 calculates its position 52 based on first relative position 50and second relative position 50. In various embodiments, device 70calculates orientation contemporaneously with calculating the positionof device 70. In various embodiments, relative position 50 andtriangulated position 50 include both the position and the orientationof device 70.

In another embodiment, lighthouse 42 emits light that is encoded withtiming data (e.g., the wavelength of the light corresponds to a time thelight is emitted), and device 70 decodes the received light to determineits position 50. In another embodiment, lighthouse 42 emits a first beamof light that rotates around lighthouse 42 in a first rotationaldirection, and lighthouse 42 also emits a second beam of light thatrotates around lighthouse 42 in a second rotational direction that isopposite the first rotational direction. In another embodiment,lighthouse 42 emits a first beam of light that rotates around lighthouse42 in a first rotational direction at a first speed, and lighthouse 42also emits a second beam of light that rotates around lighthouse 42 inthe same rotational direction but at a second speed different than thefirst speed. By timing the differences between when the different beamsof light are received by device 70, device 70 can calculate its position52. In another embodiment, device 70 can determine its position 52 basedon simultaneous localization and mapping (SLAM), in which device 70calculates its position while simultaneously mapping the area (e.g., viastereovision, triangulation, LiDAR and/or structured light projection)where device 70 is located. In alternative embodiments, other suitablemethod of determining the three-dimensional location of the device 70are used.

Projector 44 uses receiver 120 to calculate its position 52 and storesconstruction plan 56 to compare against position 52 to determine image54 to display. Projector 44 determines image 54 by identifying thelocation of projector 44 in construction plan 56. Projector 44 usesposition 52 to identify landscape 58 upon which projector 44 willdisplay image 54. Landscape 58 is the portions of the room's surfaces,such as walls, ceilings, or floors, where projector 44 will displayimage 54. Based on the identification of landscape 58, projector 44generates image 54 from construction plan 56 to display on landscape 58.Projector 44 creates object 60 in image 54 so that the location ofobject 60 corresponds to where physical object 62 will be placed orbuilt. As a result, users can look at image 54 displayed by projector 44and quickly understand and even see where physical object 62 thatcorresponds to object 60 will be built. Projector 44 may be trained toreceive indications of position 52 of corner C within room R, which maybe particularly useful in situations in which the room does not have astandard rectangular format.

Image 54 may include object 60 that is representative of physical object62 (FIG. 2 ), or image 54 may include schematic 64 of construction plan56 (FIG. 3 ). Image 54 may also include digital models of objects knownto be in the view, or scanned representations of detected objects.

Schematic 64 includes hidden object 66 that indicates where physicalobject 62 will be located behind a surface of the wall W or floor F.Schematic 64 also includes points 68 where work needs to be performed,such as where a hole needs to be drilled in the wall W or floor F.

Projector 44 may include a toggle (e.g., a button) to switch betweendifferent types of image 54 being displayed. Projector 44 displays image54 that includes object 60 representative of physical object 62 to beinstalled according to construction plan 56 (FIG. 2 ), projector 44displays image 54 that includes schematic 64 according to constructionplan 56 (FIG. 3 ), or projector 44 displays image 54 that includesoutlines of hidden objects 66 already installed or to be installed inthe floor or walls (FIG. 4 ).

Turning to FIG. 4 , system 40 may display an augmented reality (AR) byuse of AR headset 46. AR headset 46 may display augmented image 78.Augmented image 78 includes physical image 74, which is the actual viewof the room or environment, and virtual objects 60. In one embodiment ARheadset 46 displays to the user everything the user sees (e.g., a screenwith an image of the actual world overlaid with an augmented image). Inanother embodiment AR headset 46 includes a see-through portion in whichthe user directly views the actual world, and the see-through portion isa transparent display (e.g., an OLED display) with an augmented image78.

Virtual objects 60 include sinks, doors, windows, and light switches andother items that are to be installed in room R. Similar to projector 44,AR headset 46 determines its 3-D position 52, including its orientation,with respect to construction plan 56. System 10 then calculateslandscape 58 over which augmented image 78 will be displayed. In oneembodiment AR headset 46 performs some or all of the calculations, andin another embodiment a remote computing device performs some or all ofthe calculations and streams the results and/or images to AR headset 46.Based on the location and area of landscape 58, AR headset 46 identifieswhich objects 60 in construction plan 56 are within landscape 58, andthus which objects 60 should be shown in image 54. After identifyingobject 60 that will be in image 54, AR headset 46 renders arepresentation 76 of object 60 to add to physical image 74 to createaugmented image 78. AR headset 46 also optionally renders a floorrepresentation 80 of object 60 to add to physical image 74. Floorrepresentation 80 may indicate how object 60 will interact with room R,such as by indicating the area over which door object 60 will open, orby where window object 60 will be placed in wall W.

Turning to FIGS. 5-6 , system 40 may display augmented image 78 withoutreference to construction plan 56. For example, a worker createsartificial frame of reference 100, such as a Cartesian-styletwo-dimensional grid, and relies on that frame of reference 100 withoutreference to a construction plan 56. Worker calibrates a system-enabledtool 72, such as a drill, by first calibrating drill bit tip 82 thatwill be used to define artificial frame of reference 100.

In one embodiment, drill bit tip 82 is calibrated by performing severalsteps. First, tool 72 is calibrated to an environment, such as room R.As a result of the calibration, 3-D position 52, including orientation,of tool 72 within room R is known by system 40. However, position 52 ofdrill bit tip 82 (or parts and/or ends of staff 122) may not be known toa sufficient level of precision because the length and positioning ofdrill bit 84 within tool 72 may not be known (e.g., drill bit 84 may beonly partially inserted into the key of tool 72, drill bit 84 may have anon-standard length, different sized or brands of drill bits 84 may havedifferent lengths). To account for this variability in position 52 ofdrill bit tip 82 relative to tool 72, sensor 48 is placed against drillbit tip 82. System 40 calculates differential position 88 of drill bittip 82 relative to position 52 of tool 72. Subsequently, system 40 cancombine position 52 of tool 72 with differential position 88 of drillbit tip 82 to calculate position 52 of drill bit tip 82.

Once position 52 of drill bit tip 82 is calibrated, system 40 candisable tool 72 remotely to prevent damage. For example, if position 52drill bit tip 82 is approaching a delicate or dangerous object (e.g.,electrical wire, water pipe), system 40 may disable the power for tool72 before drill bit tip 82 gets closer than a threshold distance tophysical object 62.

Once drill bit tip 82 is calibrated, then a worker creates artificialframe of reference 100. First, the worker moves the drill bit tip 82 toa floor corner at the intersection between target wall TW upon whichartificial frame of reference 100 is arranged, sidewall W and floor F.The worker signals to system 40 that drill bit tip 82 is at firstcalibration point 102. The worker can signal to system by, for example,toggling a button on tool 72. First calibration point 102 is the originof the Cartesian-style grid. The worker then moves along target wall TW,places the drill bit tip 82 at a corner between target wall TW and floorF, and signals to system 40 that drill bit tip 82 is at secondcalibration point 104. System 40 combines calibration points 102 and 104to identify an axis (e.g., the X-coordinate axis) of a Cartesian-stylegrid.

In one embodiment, system 40 assumes by default that target wall TW isplumb or nearly plumb. Alternatively, the worker may identify thirdcalibration point 106 in the upper corner between target wall TW, sidewall W and ceiling C and/or fourth calibration point 108 in the oppositecorner between target wall TW, side wall W and ceiling C. Given threecalculation points 102, 104 and 106, system 10 can calculate theorientation of target wall TW, such as whether it is plumb.

After artificial frame of reference 100 is created, the worker can usesystem 40 for performing or identifying work. The worker can identifypoints 68 in wall W where holes need to be drilled. Points 68 may beidentified by positioning drill bit tip 82 at point 68 and toggling aswitch on tool 72. The worker can store drill point information 90 aboutpoints 68, such as the angle, diameter and/or depth of the hole to bedrilled at point 68. Drill point information 90 may be provided tosystem 40 by using interface 92. Optionally, system 40 may store defaultdrill point information 90 for points 68 unless the worker identifiesotherwise. After points 68 are identified, and optionally also drillpoint information 90, system 10 may include an autonomous device, suchas a drone, to perform the work identified by the worker at points 68.

As the worker navigates along wall target TW upon which artificial frameof reference 100 is situated, system 40 calculates position 52 of drillbit tip 82 as compared to artificial frame of reference 100 created bythe worker. Artificial frame of reference 100 can be a two-dimensionalgrid, as depicted in FIG. 5 , or it may be a three-dimensional grid.

The worker can use artificial frame of reference 100 instead of using alevel or tape measure. The worker reads position 52 of drill bit tip 82on interface 92 of tool 72. Once drill bit tip 82 is at the locationwhere the worker wants to mark point 68, the worker marks a first point68. The worker moves along target wall TW to place another point 68.While the worker is moving along target wall TW, interface 92 on tool 72optionally displays one or both of position 52 of drill bit tip 82 anddifferential distance 94 between drill bit tip 82 and point 68.

Based on the information provided via interface 92 of tool 72, theworker can identify points 68 on target wall TW that are plumb with eachother, level with each other, and/or equidistant with each other, andwithout need of a tape measure or level. Among other advantages, system40 combines and even improves on standard tape measures and levels toprovide workers quick and accurate measurements for points 68.

Once artificial frame of reference 100 is established, augmented image78, such as displayed in AR headset 46, may show an artificial line thatis the same distance from the ground. For example, a worker canestablish a plane by identifying at least three calibration points 102,104 and 106. Those three points define a plane, and the intersectionbetween that plane and walls or objects may be shown to users inaugmented reality, such as via AR headset 46. In another example, aworker can establish a height above the floor F with just onecalibration point 102, and a line indicating that height may be shown tousers in augmented reality, such as via AR headset 46.

Another use of artificial frame of reference 100 is to display offsets.For example, a worker identifies point 68 along a wall W, selects adistance, and the headset can display the location(s) that distance frompoint 68 (e.g., a user locates a point and 3′ and the headset showspoint(s) 68 that are 3′ from the point). This can also be used toprovide offsets, such as the worker identifying point 68 along wall W,selecting an offset distance, such as 3′, and the worker also indicateshow many additional points 68 to generate based on the offset. So if theworker requests two additional points, system 10 adds a second point 68that is offset 3′ from the first point 68, and system also adds a thirdpoint 68 another 3′ from the second point 68 along wall W, and istherefore 6′ from the first point 68. These steps may be performed tocreate virtual grids, virtual runs, and can also be mirrored or extendedonto the ceiling and/or floor.

Turning to FIG. 6 , virtual image 54 of the room and the worker can bestreamed for remote viewing. For example, position 52 of AR headset 46can be identified and then communicated to a remote computer with adisplay, such as a second AR headset 46. Second AR headset 46 receivesposition 52 and construction plan 56, and generates image 54 for asecond worker to view. As depicted in FIG. 6 , virtual image 54 is froma perspective that shows objects 60 and a virtual representation 130 ofthe first worker. This director's perspective allows second worker tosee not only what the first worker is seeing, but also what is near thefirst worker that the he/she may not see (e.g., if the first worker isclose to an object). Alternatively, virtual image 54 shown to the secondworker may be the same as seen from the first worker's perspective inroom R. Virtual image 54 may be generated based on position 52 of anyspatially-aware tool 72. For example, virtual image 54 may be based onposition 52 of tool 72, staff 122, or any other device 70.

Turning to FIG. 7 , tool 72 carried by first worker, such as AR headset46 worn by first worker, may have camera 98 that captures actual image96 that first worker is seeing. Actual image 96 includes walls W,ceiling C, floor F, and object 62. Actual image 96, such as FIG. 7 , isstreamed and displayed for a remote worker, such as via AR headset 46 orprojector 44.

Turning to FIG. 8 , augmented image 78, which is a combination ofvirtual image 54 and actual image 96 may be streamed to a remote worker.Streaming augmented image 78 allows a remote worker to see actual image96 augmented by objects 60 from construction plan 56.

Turning to FIGS. 9-11 , a worker can use system 40 to identify points68. After points 68 are identified, the same worker can go back andperform the work identified (e.g., drill the points 68), or anotherworker can have their tool 72 or AR headset 46 synced to the artificialframe of reference 100, including points 68, and subsequently performthe work identified. Point 68 includes information 90, which describescharacteristics of point 68, such as the depth of the desired holeand/or the width of the desired hole.

The worker can also identify object 60, such as window object 60 (bestshown in FIG. 10 ). The worker can trace the outline of window object 60via tool 72, staff 122, and/or any tool 72. After window object 60 isidentified and traced by the worker, window object 60 can subsequentlybe seen by the same worker or other workers.

Alternatively, the dimensions of object 60 can be traced and thedimensions subsequently used to construct a physical representation ofobject 60. For example, if the worker traces the location and dimensionof a marble countertop, the dimensions of traced object 60 can be savedand provided to a party that cuts the marble countertop to the measureddimensions. The dimensions measured and communicated could be of the2D-form and the third dimension is assumed or otherwise selected, or thedimensions measured and communicated could be of the 3D-form.

Turning to FIG. 11 , the worker can also use devices 70, such as ARheadset 46 or staff 122, to identify objects 60 to be installed in andaround room R. Worker wearing AR headset 46 views augmented image 78that includes selection menu 132, which is overlays physical image 74and the rest of augmented image 78. Selection menu 132 includescomponents 134 and component titles 136. In this example, components 134are HVAC components. The worker can select component 134 from selectionmenu 132 and identify position 52 where components 134 will be placed inroom R. After the worker identifies a component 134 to be placed in roomR, component 134 becomes virtual object 60. Worker can continue toidentify components 134 to add to augmented image 78 in room. Ascomponents 134 are added, order list 138 is updated to reflect thenumber and identity of components 134 that worker has identified areneeded. After the worker has finished completing the virtualconstruction activity, order list 138 may be submitted for order, suchas directly to a supplier or to a manager (e.g., a general contractor)for combining with other order lists 138. While constructing object 60,the worker may select different types of components other than HVACcomponents, such as plumbing components, lighting, conduit (e.g.,conduit bends), stick frame rafters, iron work, walls, sprinklersystems, datacom components, and tile work.

In one embodiment, device 70 continuously updates its position 52. Inanother embodiment, device 70 intermittently update its position 52.

Referring to FIGS. 12-22 , various tools for use with system 40 areidentified. In FIG. 12 , projector 44 on tripod 110 displays image 54with objects 60, in this case, sink objects 60. Projector 44 has height112 that can be adjusted by adjusting tripod 110. Where projector 44displays image 54 can also be adjusted by rotating projector 44 on axis114. Projector 44 includes several receivers 120 that are used tocalculate position 52 of projector 44. As projector 44 is moved aroundroom R, projector 44 updates image 54 based on construction plan 56 orartificial frame of reference 100 and points 68.

In FIG. 13 , flashlight projector 116 displays image 54. Flashlightprojector 116 calculates its position 52 based on calibration signal 150detected by receivers 120. Based on its position 52, flashlightprojector 116 generates image 54 to display. Based on construction plan56, flashlight projector 116 augments image 54 to include objects 60.Flashlight projector 116 includes several receivers 120 that are used tocalculate position 52 of projector 44. As flashlight projector 116 ismoved around room R, flashlight projector 116 updates image 54.Flashlight projector 116 includes interface 92. Interface 92 receivesinput (e.g., via an input device, such as a keyboard) and providesoutput (e.g., via an output device, such as a display) to the worker.

In FIG. 14 , lighthouse 42 includes emitter 118. Emitter 118 transmitslight, such as infrared light, to enable tools 72 to calculate theirposition 52. Various embodiments of lighthouse 42 include multipleemitters 118. For example, the position 52, timing, and direction oflight emissions from lighthouse 22 may be known by receivers 120, andrelied upon to calculate relative position 50 of receiver 120, whenreceiving from a single lighthouse 42, or position 52 of receiver 120,when receiving light from multiple lighthouses 42.

In FIG. 15 , sensor 48 includes several receivers 120. Receiver 120detects light, such as infrared light emitted by lighthouse 42, tocalculate position 52 of sensor 48. Sensor 48 may be used to calibratelocation of tool 72, such as position 52 of drill bit tip 82.

In FIG. 16 , AR headset 46 includes receivers 120 and camera 98.Receivers 120 interact with light emitters, such as lighthouses 42, todetermine position 52 of AR headset 46. Camera 98 is used to capturewhat the worker is actually viewing.

In FIG. 17 , staff 122 is used by worker to work within room R. Staff122 includes receivers 120 that are used to calculate position 52 ofstaff 122. For example, staff 122 includes a receiver 120 at handle end124 and a receiver 120 at lower end 126. Staff 122 includes interface 92that provides output and receives input from the worker. Output to aworker may include telling the worker where to go, such as viaindications on a display screen, patterns of vibrations, sounds, and/ora top view image of the worker's position with respect to their desiredlocation. Staff 122 can also be used as the marking tool to indicatepoints 68. Staff 122 may include an interface button 92 that, whendepressed, digitally marks points 68 or creates digital lines. Staff 122may provide feedback to the worker by indicating where the tip of staff122 is located. Staff 122 could also display to a worker arrows thatindicate the direction the worker should walk in. The interface 92 onstaff 122, which in some embodiments includes a display, may alsodisplay augmented image 78 in which point locations 68 are imaged withinthe physical environment as seen through the device's 70 camera.

Another way staff 122 may communicate directions to a worker is showinga hollow circle on a display, with the display representing the targetdestination. As the worker carries staff 122 around, a representation ofthe worker moves in the display. The worker's goal is to alignhimself/herself on the hollow circle.

In FIGS. 18-19 , total robotic station 140 interacts with lighthouses 42to identify position 52 of total robotic station 140. Total roboticstation 140 includes interface 92, camera 98, and tripod 110. Totalrobotic station 140 can be used to identify points 68, such as by aworker using device 70 to select point 68, or by total robotic station140 projecting a light to indicate where a previously-selected point 68is located. Total robotic station 140 can also calculate a distancebetween two locations (e.g., between point 68 and total robotic station140, between first point 68 and second point 68). Total robotic station140 can also perform 3D scanning of room R where total robotic station140 is located, and based on the scanning perform volume calculations ofroom R. Total robotic station 140 can also track devices 70, includingreceivers 120, that include prisms to reflect and/or refract light.Total robotic station 140 can also determine whether an object, such aswall W, is plumb/square, such as by scanning it.

In FIG. 20 , tripod 110 with articulating laser 142 is depicted. Lightemitted by articulating laser 142 is used to identify points 68 anchors,risers, wall locations, etc. Light emitted by articulating laser 142 caninteract with remote receivers 120 to determine position 52 of otherdevices 70. Emitter 118 emits light at mirror 162, and mirror 162adjusts its orientation to redirect where light from emitter 118 isreflected to. Articulating laser 142 includes rotary base, with a wormand spur gear, and lead screw 166, which is adjustable. Articulatinglaser 142 can determine its 3-D position 52 by emitting light in theform of a vertical line laser and scanning room R to find active laserreceivers 120 placed around the jobsite at known points 144. Oncelocated, articulating laser 142 can determine its distance to at leasttwo of these beacons by triangulating its position 52. Articulatinglaser 142 could also determine its position 52 based on learning theposition 52 of tool 72 that is mounted to articulating laser 142.

In FIG. 21 , laser receiver with vertical line laser 146 is depicted. Inone embodiment, laser receiver with vertical line laser 146 may be theQML800 available from Spectra, or a similar laser-receiving and/oremitting device.

In FIG. 22 multi-direction camera system 148 is depicted.Multi-direction camera system 148 includes a several cameras 98.

Turning to FIGS. 23A-23C, total robotic station 140 receives position 52of known points 144. Total robotic station 140 calculates its position52 based on identifying known points 144 in room R. Articulating laser142 is placed in room R. Articulating laser 142 calibrates its position52 by interacting with total robotic station 140, and then displayspoints 68 for a worker to perform actions. Articulating laser 142displays points 68 by projecting laser light on the associated position52 of ceiling C, wall W or floor F. Articulating laser 142 identifiespoints 68 within projection range 152 by referring to construction plan56 and/or artificial frame of reference 100. Projection range 152 ofarticulating laser 142 is the area over which articulating laser 142 canproject points 68. Second articulating laser 142 can be added to room R(FIG. 23C) and display points 68 after calibrating position 52 withoutreferring to total robotic station 140.

Turning to FIGS. 24A-24D, laser receiver with vertical line laser 146and tripod 110 are placed in room R. Tripod 110 may be coupled toarticulating laser 142, projector 44, or other devices 70 that projectlight in room R for workers. Tripod 110 is placed over laser receiverwith vertical line laser 146, and tripod 110 calibrates its position 52in room R, such as by identifying corners of room R and calculatingposition 52 of tripod 110 based on the relative positions of corners ofroom R (FIG. 24B). Tripod 110 has the same horizontal position 52 aslaser receiver with vertical line laser 146, so tripod 110 communicatesits horizontal position 52 to laser receiver with vertical line laser146. Tripod 110 is then moved and recalibrates its position 52 in room Rwith respect to laser receiver with vertical line laser 146 (FIG. 24C).Tripod 110 identifies points 68 within projection range 152 by referringto construction plan 56 and/or artificial frame of reference 100.Projection range 152 of tripod 110 is the area over which tripod 110 canproject points 68. Second tripod 110 can be added to room R (best shownFIG. 24D) and display points 68.

Turning to FIGS. 25A-25C, multiple sensors 48 are arranged around roomR. Tripod 110 with lighthouse 42 is placed in room R. Known points 144where sensors 48 are located are loaded into lighthouse 42. Lighthouse42 calibrates its position 52 as compared to sensors 48 at known points144 (FIG. 25B), such as by transmitting light in room R that sensor 48can sense and calculate relative position 50. Position 52 of lighthouse42 is calculated based on relative positions 50 of sensors 48.Lighthouse 42 then projects points 68 in room R (FIG. 25C).

Turning to FIGS. 26A-26C, staff 122 calibrates its position 52 tolighthouses 42 in room R. Staff 122 then identifies points 68. Afterworker moves within room R, staff 122 projects additional points 68 inroom R. Points 68 are known by staff 122 based on construction plan 56and/or artificial frame of reference 100.

Turing to FIGS. 27A-27C, staff 122 with multi-direction camera system148 calibrates its position 52 based on detecting calibration signals150 from emitters 118, such as infrared emitters 118. Staff 122 withmulti-direction camera system 148 then projects points 68 in room R.After worker moves within room R, staff 122 with multi-direction camerasystem 148 projects additional points 68 in room R. Points 68 are knownby staff 122 with multi-direction camera system 148 based onconstruction plan 56 and/or artificial frame of reference 100.

Turning to FIGS. 28A-28E, multi-direction camera system 148 projects aquickly rotating (e.g., 50 Hz, 60 Hz) light line projection 154 (bestshown in FIG. 28B). As a result, multi-direction camera system 148functions similar to lighthouse 22 in that its projection of light canbe used by receivers 120 to calculate their relative position 50. Invarious embodiments light line projection 154 is rotated bymulti-direction camera system 148 via digital rather than mechanicalmeans, providing for a more stable rotation. Based on sensors 48receiving light line projection 154, multi-direction camera system 148calibrates its position 52. Multi-direction camera system 148 projectspoints 68 in room R. After worker moves within room R, multi-directioncamera system 148 projects additional points 68 in room R. Points 68 areknown by multi-direction camera system 148 based on construction plan 56and/or artificial frame of reference 100.

Turning to FIGS. 29A-29B, projector 44 on tripod 110 calibrates itsposition 52 based on receiving calibration signals 150 from lighthouses42. Based on construction plan 56 and position 52, projector 44 displaysimage 54 that includes virtual objects 60. A worker refers to image 54and marks points 68 on a surface, such as wall W or floor F (e.g., theworker may mark an X or a circle where a hole is to be drilled for sinkobject 60). After finishing work with image 54, the worker movesprojector 44 and tripod 110 to a new location. Projector 44 recalibratesits position 52, and then projects image 54. In this embodiment,projector 44 does not continuously update its position 52, instead onlycalibrating position 52 when signaled to do so by a worker, such as theworker hitting a calibrate button on interface 92.

Turning to FIGS. 30A-30B, flashlight projector 116 calibrates itsposition 52 based on receiving calibration signals 150 from lighthouses42. Based on construction plan 56 and position 52, flashlight projector116 displays image 54 that includes virtual objects 60. A worker refersto image 54 and marks points 68 on a surface, such as wall W or floor F(e.g., the worker may mark an X or a circle where a hole is to bedrilled for sink object 60). After finishing work with image 54, theworker moves flashlight projector 116 to a new location. As flashlightprojector 116 is moved, it continuously updates its position 52 andcorrespondingly updates image 54 to reflect its new position 52.

Turning to FIG. 31 , room R includes several lighthouses 42 and knownpoints 144. Spatially aware tools 72, such as AR headset 46 and staff122, calculate their respective position 52 based on the lighthouses 42and/or known points 144. In one embodiment, tool 72 and staff 122calibrate their position 52 based on receiving calibration signals 150from lighthouses 42, and AR headset 46 calibrates its position 52 basedon recognizing known points 144 and their respective positions 52.Multiple spatially-aware devices 70 (e.g., tool 72, staff 122, ARheadset 46) are used contemporaneously in room R.

Turning to FIGS. 32A-32F, lighthouses 42 are leapfrogged across room R.Initially, lighthouses 22 are placed at known positions 52 inconstruction plan 56. Staff 122 calibrates its position 52 tolighthouses 42 and indicates points 68 on floor F. Two lighthouses 42are moved to new location 156 (FIG. 32C), and all lighthouses 42 arecalibrated to staff 122. Staff 122 calculates its position 52 based onthe two un-moved lighthouses 42, and based on that informationcalculates position 52 of lighthouses 42 at new locations 156 (FIG.21D). Position 52 of lighthouses 42 at new locations 156 is communicatedto lighthouses 42 for future calibrations. Staff 122 indicates points 68on floor F. Then, three lighthouses 42 are moved to new location 156(FIG. 32F), staff 122 calculates its position based on previously movedlighthouse 42, calculates position 52 of lighthouses 42 at new locations156 and communicates their position 52 to lighthouses 42 at newlocations 156.

In this series of images, four lighthouses 42 are leapfrogged throughroom R, with two or more lighthouses 42 relocated at each moving step.In other embodiments, room R may have any number of lighthouses 42,including only two lighthouses 42, and any subset of lighthouses 42 maybe relocated in a given moving step, including moving a singlelighthouse 42.

Turning to FIGS. 33A-33D, lighthouses 42 on tripods 110 are leapfroggedacross room R. Initially, lighthouse 42 on tripod 110 calibrates itsposition 52 with respect to known points 144. Staff 122 calibrates itsposition 52 with respect to lighthouse 42 (FIG. 33B), and identifiespoints 68. Second lighthouse 42 and second tripod 110 are calibratedwith respect to first lighthouse 42 on first tripod 110 (FIG. 33C). Inshort, second lighthouse 42 and second tripod 110 calibrates itsposition 52 indirectly with respect to known points 144 via firstlighthouse 42 on first tripod 110.

Then, first lighthouse 42 is moved to new location 156, and calibratesits new position 52 with respect to second lighthouse 42 (FIG. 33D).Lighthouses 42 may continue to leapfrog across room R, as desired. Toreduce location errors being compounded through subsequent leapfrogging,lighthouses 22 may recalibrate position 52 with respect to known points144 if/when known points 144 are available for reference.

Turning to FIG. 34 , in one embodiment AR headset 46 includes stereocameras 158 that are arranged apart from each other. Based on themarginal differences between the images received by each camera 98 instereo cameras 158, position 52 is calculated. For example, cornerbetween target wall TW, wall W, and ceiling C is at a certain locationfor each camera 98, and that location depends on the distance betweenthe corner and each camera 98. The images captured by each camera 98 cantherefore be analyzed to determine position 52 of AR headset 46 uponwhich stereo cameras 158 are situated.

Turning to FIG. 35 , physical objects 62 are not always installed inroom R exactly where construction plan 56 indicates they should beinstalled. In one embodiment image 54 projected by projector 44 is anas-built model 160. As-built model 160 reflects where the underlyingphysical objects 62 were actually installed or created when room R andassociated objects 62 were being built. As-built model 160 is generatedwhile a building is being constructed, or it may be generated after thefact based on indicating to system 400 where objects 62 exist.

In various embodiments, this disclosure describes devices 70 calculatingtheir position 52 locally based on when calibration signal 150 isreceived by receivers 120. In one embodiment, device 70 measures thetiming of when calibration signal 150 is received and communicates thattiming to a remote computer that performs the calculation. Subsequently,the results of the calculation, such as position 52, are communicatedback to device 70. In one embodiment, the timing of detected calibrationsignal 150 is communicated to a remote computer that then calculatesposition 52, and then image 54 is streamed back to device 70 and/orcommands are sent to device (e.g., disabling tool 72 if it is near a gasline).

It should be understood that the figures illustrate the exemplaryembodiments in detail, and it should be understood that the presentapplication is not limited to the details or methodology set forth inthe description or illustrated in the figures. It should also beunderstood that the terminology is for description purposes only andshould not be regarded as limiting.

Further modifications and alternative embodiments of various aspects ofthe invention will be apparent to those skilled in the art in view ofthis description. Accordingly, this description is to be construed asillustrative only. The construction and arrangements, shown in thevarious exemplary embodiments, are illustrative only. Although only afew embodiments have been described in detail in this disclosure, manymodifications are possible (e.g., variations in sizes, dimensions,structures, shapes and proportions of the various elements, values ofparameters, mounting arrangements, use of materials, colors,orientations, etc.) without materially departing from the novelteachings and advantages of the subject matter described herein. Someelements shown as integrally formed may be constructed of multiple partsor elements, the position of elements may be reversed or otherwisevaried, and the nature or number of discrete elements or positions maybe altered or varied. The order or sequence of any process, logicalalgorithm, or method steps may be varied or re-sequenced according toalternative embodiments. Other substitutions, modifications, changes andomissions may also be made in the design, operating conditions andarrangement of the various exemplary embodiments without departing fromthe scope of the present invention.

Unless otherwise expressly stated, it is in no way intended that anymethod set forth herein be construed as requiring that its steps beperformed in a specific order. Accordingly, where a method claim doesnot actually recite an order to be followed by its steps or it is nototherwise specifically stated in the claims or descriptions that thesteps are to be limited to a specific order, it is in no way intendedthat any particular order be inferred. In addition, as used herein, thearticle “a” is intended to include one or more component or element, andis not intended to be construed as meaning only one. As used herein,“rigidly coupled” refers to two components being coupled in a mannersuch that the components move together in a fixed positionalrelationship when acted upon by a force.

Various embodiments of the invention relate to any combination of any ofthe features, and any such combination of features may be claimed inthis or future applications. Any of the features, elements or componentsof any of the exemplary embodiments discussed above may be utilizedalone or in combination with any of the features, elements or componentsof any of the other embodiments discussed above.

What is claimed is:
 1. A method of tracking a construction tool within aframe of reference comprising: receiving a first indication to calculatea first location of a construction tool; calculating the first locationof the construction tool as a result of receiving the first indication;receiving a second indication to calculate a second location of theconstruction tool; calculating the second location of the constructiontool as a result of receiving the second indication; generating apositional frame of reference based on the first location and the secondlocation of the construction tool; tracking a third location and anorientation of the construction tool within the positional frame ofreference; comparing the third location and the orientation of theconstruction tool relative to the positional frame of reference;calculating relative characteristics of the construction tool based onthe comparing; generating augmentation data based on the relativecharacteristics; and providing the augmentation data to a user.
 2. Themethod of claim 1, the method further comprising: calibrating theconstruction tool before calculating the first location and the secondlocation.
 3. The method of claim 1, wherein providing the augmentationdata comprises a headset displaying an image comprising an augmentedimage based on the augmented data.
 4. The method of claim 1, whereinproviding the augmentation data comprises projecting an augmented imagebased on the augmented data.
 5. The method of claim 1 furthercomprising: receiving a plurality of indications to mark a plurality oflocations of the construction tool; generating supplemental augmentationdata based on the plurality of locations; and providing the supplementalaugmentation data to the user.
 6. The method of claim 5, the methodfurther comprising: storing drill point information about a firstlocation of the plurality of locations.
 7. The method of claim 6,wherein the drill point information is selected from the groupconsisting of an angle, a diameter, and a depth of a hole to be drilledat the first location.
 8. The method of claim 5, the method furthercomprising: displaying a distance between a first location of theplurality of locations and a location of the construction tool.
 9. Amethod of displaying information from a construction plan comprising:electronically tracking a location and an orientation of a headset beingworn by a user; calculating a spatial relationship between aconstruction plan and the location and the orientation of the headset;generating augmentation data based on the spatial relationship betweenthe construction plan and the location and the orientation of theheadset; and displaying an augmented image in the headset, the augmentedimage comprising a first image that is representative of theaugmentation data and a second image that is an actual view of theenvironment of the headset.
 10. The method of claim 9, wherein the firstimage comprises a representative image of a construction object withinthe construction plan, and wherein the construction object is selectedfrom a group consisting of a door and a window.
 11. The method of claim10, wherein the representative image is selected from a group consistingof a generic image of the construction object and a schematicrepresentation of the construction object.
 12. The method of claim 11,the method further comprising: receiving a command to toggle betweendisplaying the generic image of the construction object and displayingthe schematic representation of the construction object.
 13. The methodof claim 9, wherein the first image comprises a representative image ofa construction object within the construction plan, and wherein theconstruction object is selected from a group consisting of an applianceand an electrical outlet.
 14. The method of claim 9, wherein the firstimage comprises a representative image of a construction object withinthe construction plan, and wherein the construction object is selectedfrom a group consisting of a wall, a lighting, a conduit, a datacomcomponent, and a hole in a floor where the hole is configured to receivea pipe.
 15. The method of claim 14, wherein the representative image isselected from a group consisting of a generic image of the constructionobject and a schematic representation of the construction object. 16.The method of claim 15, the method further comprising: receiving acommand to toggle between displaying the generic image of theconstruction object and displaying the schematic representation of theconstruction object.
 17. A method of displaying information from aconstruction plan comprising: electronically tracking a location of anobject being carried by a first user; calculating a spatial relationshipbetween a construction plan and the location of the object; generatingaugmentation data based on the spatial relationship between theconstruction plan and the location of the object; and displaying anaugmented image to a second user located remotely relative to the firstuser, the augmented image comprising a first image representative ofobjects near the first user based on the augmentation data, and a secondimage representative of the first user.
 18. The method of claim 17, themethod further comprising: transmitting a signal to disable a functionof the object based on the spatial relationship between the constructionplan and the location of the object.
 19. The method of claim 17, whereinthe augmented image comprises a representative image of a constructionobject within the construction plan, and wherein the construction objectis selected from a group consisting of a door, a window, an appliance,and an electrical outlet.
 20. The method of claim 17, wherein theaugmented image comprises a representative image of a constructionobject within the construction plan, and wherein the construction objectis selected from a group consisting of a wall, a lighting, a conduit, adatacom component, and a hole in a floor where the hole is configured toreceive a pipe.